All sets of initial cliques from methods mClust, boot.sPCA, sPCA, VarCLust and BM (blockmodels) were collected on 200 scale-free graphs, and tested with the VEMtree algorithm. The true clique was also recorded and tested for each seed, and denoted as the oracle.

The questions we adress here are:

  • How each methods behaves in terms of the cliques it finds: which size, which proximity to the oracle, how many cliques found
  • Which is the performance of the clique corresponding to the maximal value of the lower bound for each method ?
  • Does the best proximity to the oracle gives the best performances ?
  • Does the best lower bound gives the best performances ?
  • For methods yielding a variable number of initial cliques (nbinit), does a high value means that the oracle is hard to find ?
  • Are failed runs (J is NaN) due to a too bad initialisation ?

All questions are studied with a special regard for the size of the oracle \(nH\) (influence of the missing actor). We define minor, medium and major influence classes as the oracle as follows:

  • minor: \(nH \leq 5\)
  • medium: \(5<nH\leq 7\)
  • major: \(nH>7\)

Data collected

influence n
Major 61
Medium 92
Minor 47

Behavior and performance

All initial cliques

Sizes and roximity to oracle (description)

What features has the set of cliques found by each method ?

Nb initial cliques
influence BM boot.sPCA mclust sPCA VarClust
Major 1.9 (0.74) 30 (14) 20 (5.3) 4 (0) 13 (0)
Medium 2.1 (0.71) 33 (12) 21 (5.4) 4 (0) 13 (0)
Minor 1.9 (0.72) 31 (13) 22 (4.4) 4 (0) 13 (0)
False Positives
influence BM boot.sPCA mclust sPCA VarClust
Major 0.63 (0.28) 0.43 (0.13) 0.21 (0.11) 0.5 (0) 0.23 (0.036)
Medium 0.56 (0.24) 0.4 (0.12) 0.28 (0.079) 0.5 (0) 0.27 (0.024)
Minor 0.63 (0.27) 0.46 (0.1) 0.3 (0.074) 0.5 (0) 0.29 (0.025)
False Negatives
influence BM boot.sPCA mclust sPCA VarClust
Major 0.38 (0.28) 0.3 (0.2) 0.65 (0.098) 0.5 (0) 0.66 (0.034)
Medium 0.47 (0.25) 0.46 (0.21) 0.69 (0.1) 0.5 (0) 0.68 (0.024)
Minor 0.38 (0.28) 0.49 (0.19) 0.74 (0.11) 0.5 (0) 0.69 (0.029)
Cliques sizes
influence BM boot.sPCA mclust sPCA VarClust
Major 8.7 (3.9) 8.6 (1.9) 4.3 (0.96) 7 (0) 4.3 (0.36)
Medium 7.6 (3.5) 6.6 (1.5) 4.1 (0.92) 7 (0) 4.1 (0.28)
Minor 8.8 (3.9) 6.7 (1.1) 4 (0.73) 7 (0) 4.1 (0.28)

Performances

What performances has the set of cliques found by each method ?

AUC
influence BM boot.sPCA mclust sPCA VarClust
Major 0.89 (0.11) 0.9 (0.12) 0.77 (0.073) 0.78 (0.097) 0.75 (0.043)
Medium 0.85 (0.1) 0.81 (0.1) 0.77 (0.049) 0.78 (0.085) 0.77 (0.037)
Minor 0.84 (0.08) 0.79 (0.076) 0.79 (0.042) 0.8 (0.048) 0.8 (0.035)
PPVH
influence BM boot.sPCA mclust sPCA VarClust
Major 0.86 (0.17) 0.84 (0.2) 0.77 (0.15) 0.68 (0.16) 0.77 (0.11)
Medium 0.68 (0.23) 0.6 (0.23) 0.54 (0.13) 0.52 (0.18) 0.59 (0.11)
Minor 0.45 (0.23) 0.4 (0.2) 0.38 (0.14) 0.42 (0.14) 0.47 (0.14)
TPRH
influence BM boot.sPCA mclust sPCA VarClust
Major 0.69 (0.27) 0.75 (0.24) 0.37 (0.16) 0.57 (0.16) 0.35 (0.084)
Medium 0.5 (0.28) 0.51 (0.25) 0.27 (0.12) 0.45 (0.19) 0.29 (0.08)
Minor 0.41 (0.29) 0.4 (0.23) 0.21 (0.11) 0.42 (0.16) 0.26 (0.086)

Selection with best J

Sizes and proximity to oracle

What feature has the clique corresponding to the maximal value of the lower bound for each method ?

False Positives
influence BM boot.sPCA mclust sPCA VarClust
Major 0.56 (0.4) 0.3 (0.24) 0.22 (0.23) 0.37 (0.26) 0.37 (0.3)
Medium 0.63 (0.31) 0.44 (0.22) 0.33 (0.19) 0.49 (0.22) 0.47 (0.28)
Minor 0.7 (0.28) 0.48 (0.17) 0.34 (0.21) 0.52 (0.2) 0.4 (0.33)
False Negatives
influence BM boot.sPCA mclust sPCA VarClust
Major 0.28 (0.27) 0.2 (0.21) 0.48 (0.24) 0.14 (0.15) 0.23 (0.26)
Medium 0.41 (0.3) 0.32 (0.32) 0.55 (0.3) 0.38 (0.34) 0.47 (0.37)
Minor 0.38 (0.34) 0.42 (0.3) 0.68 (0.27) 0.39 (0.33) 0.53 (0.34)
Cliques sizes
influence BM boot.sPCA mclust sPCA VarClust
Major 9.3 (4) 8.8 (2.3) 5.9 (1.8) 9.7 (1.6) 9 (2.8)
Medium 8.6 (3.6) 7.8 (2.8) 5.4 (2.1) 7.7 (2) 7 (3.6)
Minor 9.3 (3.9) 7.2 (2.4) 4.7 (1.8) 7.7 (1.8) 6 (4)

Performances

What is the performance of the clique corresponding to the maximal value of the lower bound for each method ?

AUC
influence BM boot.sPCA mclust sPCA VarClust
Major 0.95 (0.076) 0.96 (0.11) 0.89 (0.15) 0.98 (0.085) 0.95 (0.12)
Medium 0.88 (0.12) 0.88 (0.15) 0.83 (0.15) 0.84 (0.16) 0.84 (0.16)
Minor 0.84 (0.09) 0.81 (0.1) 0.82 (0.088) 0.84 (0.11) 0.84 (0.1)
PPVH
influence BM boot.sPCA mclust sPCA VarClust
Major 0.94 (0.14) 0.93 (0.19) 0.84 (0.28) 0.95 (0.16) 0.91 (0.23)
Medium 0.72 (0.28) 0.73 (0.33) 0.62 (0.33) 0.64 (0.33) 0.62 (0.35)
Minor 0.43 (0.27) 0.43 (0.28) 0.42 (0.29) 0.49 (0.28) 0.52 (0.29)
TPRH
influence BM boot.sPCA mclust sPCA VarClust
Major 0.82 (0.24) 0.9 (0.24) 0.67 (0.34) 0.91 (0.19) 0.85 (0.28)
Medium 0.6 (0.31) 0.7 (0.37) 0.5 (0.34) 0.61 (0.36) 0.55 (0.38)
Minor 0.43 (0.32) 0.43 (0.32) 0.32 (0.25) 0.54 (0.35) 0.43 (0.31)

Proximity and global performance

FN seems to control all performances.

PPVH

TPRH

AUC/FP

Proximity and performance of best FN

We filter all seeds by their minimum FN reached, for each method and influence.

PPVH

TPRH

AUC

Proximity and performance of best J

We filter all seeds by their maximum lower bound value J reached, for each method and influence.

PPVH

TPRH

AUC

Failed runs

mJnan
influence BM boot.sPCA mclust Oracle sPCA VarClust
Major 0.40 0.2 0.17 NA 0.50 0.24
Medium 0.55 0.3 0.30 1 0.48 0.28
Minor 0.70 0.4 0.36 1 0.53 0.40
FP
influence BM boot.sPCA mclust Oracle sPCA VarClust
Major 0.65 (0.16) 0.42 (0.14) 0.22 (0.15) NA 0.52 (0.18) 0.2 (0.14)
Medium 0.37 (0.23) 0.42 (0.14) 0.28 (0.12) 0 (0) 0.51 (0.18) 0.23 (0.11)
Minor 0.4 (0.21) 0.46 (0.12) 0.3 (0.094) 0 (0) 0.48 (0.14) 0.27 (0.097)
FN
influence BM boot.sPCA mclust Oracle sPCA VarClust
Major 0.69 (0.23) 0.55 (0.16) 0.73 (0.12) NA 0.73 (0.23) 0.78 (0.12)
Medium 0.63 (0.25) 0.58 (0.19) 0.73 (0.16) 0 (0) 0.58 (0.29) 0.75 (0.15)
Minor 0.52 (0.25) 0.5 (0.18) 0.75 (0.15) 0 (0) 0.53 (0.22) 0.75 (0.13)

Filtering by failed runs, we observe:

other viz